Automatic method to delineate or categorize an electrocardiogram

ABSTRACT

Disclosed is a method for computerizing delineation and/or multi-label classification of an ECG signal, including: applying a neural network to the ECG, labelling the ECG, and optionally displaying the labels according to time with the ECG signal.

FIELD OF INVENTION

The present invention relates to temporal signal analysis, preferablycardiac signal analysis, more preferably electrocardiogram analysis,using at least one neural network.

BACKGROUND OF INVENTION

Electrocardiogram (ECG) and endocardiogram are graphic representationsof the electrical activity of the heart. Electrocardiogram is recordedfrom the body using a number of electrodes placed in specific predefinedareas. It is considered as a fundamental tool of clinical practice. Itis a simple, non-invasive exam that can be performed by any healthprofessional. Placing the electrodes is not considered as a medicalprocedure, yet in some countries, the prescription of the ECG by adoctor is essential for it to be performed. It is known that the ECGconstitutes the first step in cardiovascular diseases (CVD) diagnosis,and is used multiple times throughout the life of a CVD patient. CVDconstitute the first global cause of death.

A cardiac signal is composed of one or multiple synchronized temporalsignals, called lead signals. The ECG shown in FIG. 1 represents astandard 12-lead resting ECG, with its 12 standard deviations recordingduring 10 seconds. Some ECG, specifically known as Holters, may recordonly one lead for a period of time which can be of more than 7 days.

A cardiac signal displays repeating patterns usually comprising aP-wave, a QRS complex and a T-wave, respectively corresponding to thedepolarization of the atria, depolarization of the ventricles andrepolarization of the ventricles. These waves and complexes are shown inFIG. 2, which focuses on a couple of beats in one lead signal.

Cardiac signals allow for the detection of many abnormalities, whichoften in turn point to specific CVD. It is estimated that about 150measurable abnormalities can be identified on an ECG recordings today.However, without specific expertise and/or regular training, only asmall portion of these abnormalities can be easily spotted.Unfortunately, today, it is estimated that only one third of ECGs areperformed in settings where cardiology expertise is readily available.

In order to make cardiac signal interpretation, especially ECGinterpretation simpler and assist non-specialists, two alternativesexist today, but neither fully satisfy the needs of healthprofessionals:

-   -   Telecardiology centers, where an interpretation of an ECG sent        by a non-specialist is delivered either by a cardiologist or by        a specialized ECG technician. Their interpretations are of high        quality but are slow and expensive to obtain.    -   Prior art automated cardiac signal interpretation softwares,        which are mainly developed by cardiac signal device        manufacturers. They provide low quality interpretation (false        alarms are very frequent) but deliver them in seconds.

Prior art automated cardiac signal interpretation softwares can providetwo types of information about a cardiac signal:

-   -   a local information called delineation, providing the temporal        location of each wave and optionally qualifying each wave        separately; and/or    -   a global information providing a classification of the cardiac        signal as normal/abnormal or labeling its abnormalities.

Concerning delineation, two main approaches are used for finding thewaves of cardiac signals.

The first one is based on multiscale wavelet analysis. This approachlooks for wavelet coefficients reaching predefined thresholds atwell-chosen scales (Martinez et al, IEEE transactions on biomedicalengineering, Vol. 51, No. 4, April 2004, 570-581, Almeida et al., IEEEtransactions on biomedical engineering, Vol. 56. No. 8, August 2009, pp1996-2005. Boichat et al., Proceedings of Wearable and Implantable BodySensor Networks, 2009, pp 256-261, U.S. Pat. No. 8,903,479, 2014-12-02,Zoicas et al.). The usual process is to look for QRS complexes, and thenlook for P waves on the signal before the complexes, and after them forT waves. This approach can only handle a single lead at a time,sometimes using projection to one artificial lead (US2014/0148714-2014-05-29, Mamaghanian et al.). This computation is madevery unstable by the use of thresholds. The approach is also limited asit can neither deal with multiple P waves nor with “hidden” P waves.

The second one is based on Hidden Markov Models (HMM). This machinelearning approach considers that the current state of the signal(whether a sample is either part of a QRS complex, a P wave, a T wave orno wave) is a hidden variable that one wants to recover (Coast et al.,IEEE transactions on biomedical engineering. Vol. 37, No. 9, September1990, pp 826-836, Hughes et al., Proceedings of Neural InformationProcessing Systems, 2004, pp 611-618, U.S. Pat. No. 8,332,017,2012-12-11. Trassenko et al.). To this end, a representation of thesignal must be designed using handcrafted “features”, and a mathematicalmodel must be fitted for each wave, based on these features. Based on asufficient number of examples, the algorithms can learn to recognizeeach wave. This process can however be cumbersome since the featuredesign is not straightforward, and the model, usually Gaussian, is notwell adapted. Also, none of these works has considered the situation ofhidden P waves.

In the state-of-the-art, characterization of the waves for thedelineation is only performed on the QRS to detect for instanceventricular or paced beats, and done in a second step, once the waveshave already been localized. Such methods usually use standardclassification algorithms which learn the type of beat based on manytraining examples of handcrafted set of features and corresponding beatlabel (Chazal et al., IEEE Transactions on Biomedical Engineering, 2004,vol. 51, pp. 1196-1206). These methods are limited in that the featureswhich have been handcrafted will always be suboptimal since they werenot learnt and may have erased some crucial information.

In order to solve the above issues, the latest works (Kiranyaz et al,IEEE Transactions on Biomedical Engineering, 2016, Vol. 63, pp 664-675)have turned to novel architectures called neural networks which havebeen intensively studied and had great results in the field of imaging(Russakovsky et al., arXiv:1409.0575v3, 30 Jan. 2015). Indeed, thesemethods bypass the need of handcrafted features and directly learn fromraw or mildly preprocessed data. Still, these applications of neuralnetworks to cardiac signal waves characterization are very limitedsince:

-   -   they must first rely on an algorithm able to detect the waves;    -   they were only developed for QRS characterization; and    -   they lack context information in processing one beat at a time,        surrounding beats often providing important information.

Concerning abnormalities and/or CVD detection, most algorithms use rulesbased on temporal and morphological indicators computed using thedelineation: PR, RR and QT intervals. QRS width, level of the STsegment, slope of the T wave, etc. . . . . These rules such as theMinnesota Code (Prineas et al., Springer, ISBN 978-1-84882-777-6, 2009)were written by cardiologists. However, they do not reflect the way thecardiologists analyze the ECGs and are crude simplifications. Algorithmssuch as the Glasgow University Algorithm are based on such principles(Statement of Validation and Accuracy for the Glasgow 12-Lead ECGAnalysis Program. Physio Control, 2009).

More advanced methods use learning algorithms, and are built using adiagnosis and an adequate representation for each cardiac signal theylearn from. In, Shen et al., Biomedical Engineering and Informatics(BMEI), 2010, vol. 3, pp. 960-964 for instance, the author used supportvector machines to detect bundle branch blocks. However, in thesemethods, once again, it is necessary to seek a representation of the rawdata into a space that preserves the invariance and stabilityproperties. Indeed, cardiac signals vary significantly from one patientto another. It is therefore extremely difficult for an algorithm tolearn how to discriminate different diseases by simply comparing rawdata. A representation which drastically limits this interpatientvariability while preserving the invariance within the same diseaseclass must be chosen. Also, once again these representations usuallyrely on a preliminary detection of the beats and hence in a reliabledelineation.

Some scientific teams very recently also turned to neural networkarchitectures, but limitations still arose when they attempted to applythem to ECGs.

One team (Jin and Dong, Science China Press, Vol. 45, No 3, 2015, pp398-416; CN104970789) proposed binary classification on a full ECG,hence providing one and only one class for any analyzed ECG. This is forinstance a classification normal Vs abnormal (see [0027] ofCN104970789). Their architecture use convolutional layers which processthe leads independently before mixing them into fully connected layers.The authors also mention multi-class analysis, aiming at recovering oneclass among several, but they do not consider the less commonly usedmulti-label classification, which is however crucial in ECG analysissince one ECG can have several abnormalities such as for instance a leftbundle branch block with atrial fibrillations.

Thus, there is a need for methods able to analyze cardiac signal,especially ECG, that can:

-   -   carry out the analysis without the need for beat-by-beat        processing, or feature extraction;    -   obtain the delineation of the signal, including identification        of hidden P waves, qualification of each wave in a single step,        and optionally present this information in an comprehensible        way;    -   provide a multi-label classification directly from at least one        time window of a cardiac signal, generally exhibiting multiple        labels, contrary the prior art, which provides a single        exclusive label;    -   process data with varying number of leads with a same neural        network;    -   be fast, stable and reliable.

SUMMARY

To address the above issues in cardiac signal analyses, the Applicantdeveloped two techniques based on convolutional neural networks:

-   -   A convolutional neural network which first gives a dense        prediction of the probability of presence of each wave on each        time stamp of the cardiac signal, then post-processes the signal        to produce its delineation. This novel approach of the        delineation, using convolutional networks, allows the processing        of cardiac signals of any duration, analyzing and qualifying all        types of waves in the same way in a single step, without being        constrained by their positions.    -   A convolutional neural network which directly predicts multiple        labels on the cardiac signal. It results in a fixed format        multi-label output which can represent abnormalities such as for        instance “Atrial fibrillations” or descriptors such as for        instance “Normal sinus rhythm” or “Noisy ECG”.

Thus, the present invention relates to a method for computerizing thedelineation of a cardiac signal comprising a plurality of time points,said method comprising: applying a convolutional neural network NN1 tosaid cardiac signal, whereby the convolutional neural network NN1 readseach time point of the cardiac signal, analyzes temporally each timepoint of the cardiac signal, assigns to each time point of the cardiacsignal a score for at least one wave among the following waves: P-wave,QRS complex, T-wave.

According to one embodiment, the convolutional neural network NN1assigns to each time point of the cardiac signal a score for at leastthe following waves: P-wave, QRS complex, T-wave. According to oneembodiment, the convolutional neural network assigns to each time pointof the cardiac signal a score for the hidden P waves. According to oneembodiment, the convolutional neural network NN1 is a fullyconvolutional neural network.

According to an embodiment, the method further comprises a pre-treatmentstep, wherein the pre-treatment comprises denoising and removing thebaseline of the cardiac signal as well as expressing it at a chosenfrequency prior to the application of NN1.

According to an embodiment, the method further comprises apost-treatment step computing the time points of the beginning and theend of each wave in the cardiac signal, called the onset and the offset,and other information such as for instance prematurity, conduction andorigin of the waves. According to one embodiment, the method furthercomprises a post-treatment step computing global or local measurementsbased on the onset and the offset of each wave and the signal, such asfor instance PR interval, ST elevation and heart rate. According to oneembodiment, the method further comprises a post-treatment step computingdelineation-based labels based on the global or local measurements.

According to one embodiment, the convolutional neural network NN1 isable to process a cardiac signal recorded from any number of leads.

The invention also comprises a software comprising a trained neuralnetwork for delineation of a cardiac signal. The invention alsocomprises a computer device comprising a software implementing a methodfor delineation of a cardiac signal, comprising applying a convolutionalneural network NN1 to said cardiac signal, as described above. Accordingto one embodiment, the computer device further comprises a displayconfigured for displaying the wave locations and optionallysimultaneously the cardiac signal and/or an application programminginterface for recovering the delineation-based labels and/or delineationfor any given cardiac signal.

This invention also includes a method for computerizing multi-labelclassification of a cardiac signal having a plurality of time points,comprising applying a convolutional neural network NN2 to said cardiacsignal, whereby the convolutional neural network NN2 reads each timepoint of the cardiac signal, analyzes each time point of the cardiacsignal, computes scores on a time window aggregating at least two timepoints for a plurality of predetermined non-exclusive labels, such asfor instance normal cardiac signal, artefact or atrial fibrillation, andallots to the time window the labels which have a score higher than atleast one predetermined threshold.

According to one embodiment, the convolutional neural network NN2 is arecurrent neural network.

According to an embodiment, the method further comprises a pre-treatmentstep, wherein the pre-treatment comprises denoising and removing thebaseline of the cardiac signal as well as expressing it at a chosenfrequency prior to the application of NN2.

According to an embodiment, the method further comprises apost-treatment step, comprising a filtering step so as to removeredundant labels, and optionally incorporating delineation-derivedlabels such as for instance first degree atrioventricular block (long PRinterval), and optionally computing the onset and offset times of eachabnormality. According to one embodiment, the convolutional neuralnetwork NN2 is able to process a cardiac signal recorded from any numberof leads.

The invention also comprises a software comprising a trained neuralnetwork for multi-label classification of a cardiac signal. Theinvention also comprises a computer device comprising a softwareimplementing a method for multi-label classification of a cardiacsignal, comprising applying a recurrent neural network NN2 to saidcardiac signal, as described above. According to one embodiment, thecomputer device further comprises a display configured for displayingthe scores of the labels which have been allotted to a time window andoptionally simultaneously the cardiac signal; and/or an applicationprogramming interface for recovering the labels.

Furthermore, the invention also concerns a method for computerizingmulti-label classification of a cardiac signal, having a plurality oftime points, said method comprising applying a convolutional neuralnetwork NN1 to said cardiac signal, wherein the neural network: readseach time point of the cardiac signal, analyzes temporally each timepoint of the cardiac signal, assigns to each time point of the cardiacsignal a score for at least the following waves: P-wave, QRS complex,T-wave; computes the onset and the offset of each wave in the cardiacsignal based on the scores assigned to each time point; computes globalmeasurements based on the onset and the offset of each wave; andapplying a convolutional neural network NN2 to said cardiac signal,wherein the neural network: reads each time point of the cardiac signaland the global measurements obtained from NN1, analyzes each time pointof the cardiac signal and the global measurements obtained from NN,computes scores on a time window aggregating at least two time pointsfor a plurality of predetermined non-exclusive labels, such as forexample normal cardiac signal, artefact or, atrial fibrillation, allotsto the time window the labels which have a score higher than thepredetermined threshold.

According to one embodiment, the method further comprises apre-treatment step, wherein the pre-treatment comprises denoising andremoving the baseline of the cardiac signal as well as expressing it ata chosen frequency prior to the application of NN1 and NN2. According toone embodiment, the method further comprises a post-treatment stepcomputing delineation-based labels, removing redundant labels, andoptionally computing onset and offset of each abnormality. According toone embodiment, the convolutional neural networks are able to process acardiac signal recorded from any number of leads. The invention alsocomprises a software comprising a trained neural network for delineationof a cardiac signal. The invention also comprises a computer devicecomprising a software implementing said method, comprising applyingconvolutional neural networks NN1 and NN2 to said cardiac signal, asdescribed above. According to one embodiment, the computer devicefurther comprises a display configured for displaying the wavelocations, the scores of the labels which have been allotted to a timewindow and optionally simultaneously the cardiac signal; and/or anapplication programming interface for recovering the labels and/ordelineation for any given cardiac signal.

Furthermore, the invention also includes a method for computerizingdelineation and multi-label classification of a cardiac signal having aplurality of time points, comprising applying a trained neural networkNN3 to said cardiac signal, whereby the recurrent neural network NN3reads each time point of the cardiac signal, analyzes temporally eachtime point of the cardiac signal, assigns to each time point of thecardiac signal a score for at least the following waves: P-wave, QRScomplex, T-wave; computes scores on a time window aggregating at leasttwo time points for a plurality of predetermined non-exclusive labels,such as for example normal cardiac signal, artefact or atrialfibrillation; and allots to the time window the labels which have ascore higher than at least one predetermined threshold.

According to an embodiment, the method further comprises a pre-treatmentstep, wherein the pre-treatment comprises denoising and removing thebaseline of the cardiac signal as well as expressing it at a chosenfrequency prior to the application of NN3.

According to one embodiment, the method further comprises apost-treatment step computing the onset and offset of each wave in thecardiac signal optionally with other information such as for instanceprematurity, conduction and origin of the waves; computingdelineation-derived labels; removing redundant labels, and optionallyproducing onset and offset of each abnormality and global and localmeasurements such as for instance PR interval and heart rate.

The invention also comprises a software comprising a trained neuralnetwork for delineation and multi-label classification of a cardiacsignal. The invention also comprises a computer device comprising asoftware implementing a method for delineation and multi-labelclassification of a cardiac signal, comprising applying a neural networkNN3 to said cardiac signal, as described above. According to oneembodiment, the computer device further comprising a display configuredfor displaying the wave locations, the scores of the labels which havebeen allotted to a time window and optionally simultaneously the cardiacsignal; and/or an application programming interface for recovering thelabels and/or delineation for any given cardiac signal.

Definition

“Abnormality” refers to any physiological abnormality which can beidentifiable on the cardiac signal. Today about 150 measurableabnormalities can be identified on cardiac signal recordings. Forinstance, within the present invention, the following abnormalities maybe nonlimitatively identified: “Sinoatrial block, paralysis or arrest”,“Atrial Fibrillation”, “Atrial fibrillation or flutter”, “AtrialFlutter”, “Atrial tachycardia”, “Junctional tachycardia”,“Supraventricular tachycardia”, “Sinus tachycardia”, “Ventriculartachycardia”. “Pacemaker”, “Premature ventricular complex”, “Prematureatrial complex”, “First degree atrio-ventricular block (AVB)”, “2^(nd)degree AVB Mobitz I”, “2^(nd) degree AVB Mobitz II”, “3^(rd) degreeAVB”, “Wolff-Parkinson-White syndrome”, “Left bundle branch block”,“Right bundle branch block”, “Intraventricular conduction delay”, “Leftventricular hypertrophy”, “Right ventricular hypertrophy”, “Acutemyocardial infarction”, “Old myocardial infarction”, “Ischemia”,“Hyperkalemia”, “Hypokalemia”, “Brugada”, “Long QTc”, etc. . . . .

“Cardiac signal” refers to the signal recording the electricalconduction in the heart. Said cardiac signal may be for instance anelectrocardiogram (ECG) or an endocardiogram. Such signals may have oneor more channels, called leads. It may be short term (10 seconds instandard ECGs) or long term (several days in Holters).

“Classification” refers to the task of categorizing objects into a listof groups. Such a task includes for instance recognizing the animal froma picture (the list of groups is then a list of animals), or recognizingwhether an ECG is normal or abnormal.

“Multi-label classification” refers to identifying objects as being partof none, one or several groups of a given list of groups. Such a taskincludes for instance identifying none to several animals from apicture, or identifying none to several abnormalities on an ECG.

“Delineation” refers to the identification of the temporal localizationof each of the waves of a cardiac signal. Delineation can alsooptionally provide more precise characterization of each of the waves.

“Descriptor” refers to a description of a cardiac signal which is not anabnormality, such as for instance “Normal ECG”, “Normal sinus rhythm” or“Noisy cardiac signal”, “Electrode inversion”, etc. . . . .

“Hidden P wave” refers to a P wave which occurs during another wave orcomplex, such as for example during a T wave.

“Label” refers to a class used within the present invention formulti-label classification of a cardiac signal. Said label can be anabnormality or a descriptor. Labels are none exclusive. For instance,one can observe an Atrial fibrillation and Wolff-Parkinson-Whitetogether.

“Delineation-based labels” refers to labels which can be deduced (i.e.computed) from the delineation and its measurements. For instance,within the present invention, the following delineation-based labels maybe nonlimitatively: “short PR interval” (PR interval<120 ms), “Firstdegree AV block” (PR interval>200 ms), axis deviations, “Long QTc”,“Short QTc”, “Wide complex tachycardia”, intraventricular conductionblocks, etc. . . . .

“Local measurements” refers to measurements directly derived from thedelineation, such as for instance a given RR interval (duration betweenone QRS complex and the following).

“Global measurements” refers to measurements derived from thedelineation and aggregated through time, such as for instance a mean ormedian values of PR interval (duration between the beginning of aconducted P wave and the following QRS complex). P duration, QRSduration, QRS axis, median QT interval, corrected QT interval (Qtc),corrected JT interval, heart rate, ST elevation. Sokolov index, numberof premature ventricular complex, number of premature atrial complexes,ratio of non-conducted P waves, ratio of paced waves etc. . . . .

“Neural network” refers to a mathematical structure taking an object asinput and producing another object as output though a set of linear andnon-linear operations called layers. Such structures have parameterswhich can be tuned through a learning phase so as to produce aparticular output, and are for instance used for classificationpurposes. The input is then the object to categorize, and the output theprobabilities to pertain in each of the categories.

“Convolutional neural network” refers to a neural network which ispartly composed of convolutional layers, i.e. layers which apply aconvolution on their input.

“Fully convolutional neural network” refers to a convolutional neuralnetwork in which all linear operations are convolutions.

“Recurrent convolutional neural network” refers to a particularconvolutional neural network structure able to keep a memory on theprevious objects it has been applied to.

“Lead invariant structure” refers to a structure proposed by theapplicant to be able to use a same neural network for signals with anynumber of channels. Said structure is preferably used for neuralnetworks processing Holters but not for networks processing standard 12lead ECGs.

DETAILED DESCRIPTION

The present invention relates to temporal signal analysis, preferablycardiac signal analysis, using at least one convolutional neuralnetwork.

According to one embodiment, the cardiac signal is recorded from anynumber of leads during from 1 second to several days.

According to one embodiment, the cardiac signal is recorded from 12leads or more. According to an alternative embodiment, the cardiacsignal is recorded from strictly less than 12 leads.

According to one embodiment, the cardiac signal is recorded from 12leads or more under direct medical supervision (resting ECG, stresstest, etc.).

According to an alternative embodiment, the cardiac signal is recordedfrom strictly less than 12 leads or not under direct medical supervision(ambulatory monitoring, etc.).

The framework used here is the one of supervised learning. The aim ofsupervised learning is to predict an output vector Y from an inputvector X. In the Applicant embodiment, X is a cardiac signal (amultivariate signal) as a matrix of size m×n. As for Y, in the Applicantembodiment, it can be:

-   -   the delineation, providing a score for each sample of X to be        part of one of the different waves as a matrix of size p×n;    -   the scores for each label as a vector of size q;    -   the set composed of both the delineation and the vector of        scores.

The problem of supervised learning can also be stated as follows:designing a function f such that for any input X, f(X)≈Y. To this end,the function f is parametrized, and these parameters are “learned”(parameters are optimized with regards to an objective loss function,for example, by means of a gradient descent (Bishop, Pattern Recognitionand Machine Learning, Springer, 2006, ISBN-10: 0-387-31073-8).

A neural network is a particular type of function f, aiming at mimickingthe way biological neurons work. One of the most basic and earliestneural network is the perceptron (Rosenblatt. Psychological Review, Vol.65, No. 6, 1958, pp 386-408). From the input X, it computes linearcombinations (i.e. weighted sums) of the elements of X through amultiplication with a matrix W, adds an offset b, and then applies anon-linear function σ, such as for instance a sigmoid, on every elementof the output:

f(X)=σ(WX+B)

The parameters which are learned in a perceptron are both W and B. Inpractice, more general neural networks are just compositions ofperceptrons:

f(X)=σ_(n)(W _(n) . . . σ_(n)(W ₁ X+B ₁)+B _(n))

The output of a perceptron can be sent as input to another one. Theinput, the final output, and the intermediate states are called layers.The intermediate ones are more specifically called hidden layers, sinceonly the input and the final output are observed. For instance, a neuralnetwork with one hidden layer can be written as:

f(X)=σ₂(W ₂σ₁(W ₁ X+B ₁)+B ₂)

Such a network is shown in a graphic form as an example in FIG. 3. Thevector X enters the network as the input layer, each element of thehidden layer is then computed from linear combinations of all elementsof X (hence all the links), and the element of the output layer are thencomputed from linear combinations of all elements of the hidden layer.

It has been shown that neural networks in their general form are able toapproximate all kinds of functions (Cybenko, Math. Control SignalsSystems, Vol. 2, 1989, 20 pp 303-314). The term “deep learning” is usedwhen a neural network is composed of many layers (though the thresholdis not perfectly defined, it can be set to about ten). This field arosemostly in the last decade, thanks to recent advances in algorithms andin computation power.

Convolutional neural networks are a particular type of neural networks,where one or more of the matrices W; which are learned do not encode afull linear combination of the input elements, but the same local linearcombination at all the elements of a structured signal such as forexample an image or, in this specific context, a cardiac signal, througha convolution (Fukushima, Biol. Cybernetics. Vol. 36, 1980, pp 193-202,LeCun et al., Neural Computation, Vol. 1, 1989, pp 541-551). Anillustration of a convolutional neural network is shown in FIG. 6. Mostconvolutional neural networks implement a few convolutional layers andthen standard layers so as to provide a classification. A network whichonly contains convolutional networks is called a fully convolutionalneural network. Finally, a recurrent convolutional neural network is anetwork composed of two sub-networks: a convolutional neural networkwhich extracts features and is computed at all time points of thecardiac signal, and a neural network on top of it which accumulatesthrough time the outputs of the convolutional neural network in order toprovide a refined output. An illustration of a recurrent convolutionalneural network is provided in FIG. 7.

As mentioned above, a cardiac signal, especially an ECG is representedas a matrix of real numbers, of size m×n. The constant m is the numberof leads, typically 12, though networks can be taught to process cardiacsignal with any number of leads, as detailed herebelow. The number ofsamples n provides the duration of the cardiac signal n/f, with f beingthe sampling frequency of the cardiac signal. A network is trained for agiven frequency, such as for example 250 Hz or 500 Hz or 1000 Hz, thoughany frequency could be used. A same network can however process cardiacsignal of any length n, if it is fully convolutional or a recurrentneural network.

In both the delineation and the multi-label classification embodiments,networks are expressed using open softwares such as for exampleTensorflow, Theano, Caffe or Torch. These tools provide functions forcomputing the output(s) of the networks and for updating theirparameters through gradient descent. The exact structure of the networkis not extremely important. Preferred choices are fully convolutionalnetworks in the situation of the delineation network (Long et al.,Proceedings of Computer Vision and Pattern Recognition, 2015, pp3431-3440), convolutional (Krizhevsk et al., Proceedings of NeuralInformation Processing Systems, 2012, pp 1097-1105) in the situation ofthe multi-label classification network, or recurrent neural networks(Donahue et al., arXiv:1411.4389v3, 17 Feb. 2015 and Mnih et al.,arXiv:1406.6247v1, 24 Jun. 2014) for both the multi-label classificationnetwork and the delineation network. The 2D convolutional layers whichwere used on images are then easily converted into 1D convolutionallayers in order to process cardiac signals.

In one embodiment, the network is amended to process data with varyingnumber of leads in entry. In one embodiment, the neural network furthercomprises a sequence of layers at the beginning of the network so as toobtain a network which is independent of the number of input leads andcan therefore process cardiac signals with any number of leads m. Such astructure is presented in FIG. 8 with m=2 input leads and k=3 outputsignals. The same structure can process any number of input leads m andwill still provide k=3 signals in output, which can be fed to the restof the network for which a fixed number of input signals is required. Inthis way, m need not be fixed anymore. According to one embodiment, inorder to obtain a k-lead signal from an m-lead cardiac signal, the mleads are convoluted using a lead-by-lead convolution with k filters,the signal are then grouped by convolution filter in order to obtain kgroups of m leads and a mathematical function is finally apply to eachgroup to obtain k leads. According to one embodiment, any number ofoutputs k can be chosen. According to one embodiment, any number ofinputs m can be used. According to one embodiment, the mathematicalfunction is the maximum at each time point or may be any other functionknown to one skilled in the art. According to the Applicant, thisfeature was never disclosed before.

This invention also pertains to a method for manufacturing a neuralnetwork for delineation of a cardiac signal, by training it.

The training phase of the neural networks in the embodiment ofdelineation consists in the following steps:

-   -   taking one cardiac signal from a dataset containing cardiac        signals and their known delineation; the cardiac signal being        expressed as a matrix of size m×n with m fixed and at a        predefined frequency:    -   expressing the delineation of this cardiac signal under the form        of a matrix y of size p×n where p is the number of annotated        types of wave; typically p=3, so as to identify P waves, QRS        complexes, and T waves; annotations are expressed as lists of        wave with their start and end points such as for example: (P,        1.2s, 1.3s), (QRS 1.4s 1.7s), (T, 1.7, 2.1), (P, 2.2, 2.3); in        this example, the first row of y, corresponding to P waves, will        be 1 for samples corresponding to times between 1.2 and 1.3s,        and between 2.2 and 2.4s, and 0 otherwise; row 2 will correspond        to QRS complexes and row 3 to T waves;    -   computing the output of the network for this cardiac signal;    -   modifying the parameters of the network so as to decrease a cost        function comparing the known delineation and the output of the        network; a cross-entropy error function is used so as to allow        for multi-labeling (allowing for multiple waves at a given        instant); this minimization can be done though a gradient step    -   repeating steps 1 to 4 at least once for each cardiac signal of        the dataset;    -   recovering the neural network.

According to one embodiment, delineation further comprises wavecharacterization. According to said embodiment, p is the number ofannotated types of wave plus the number of wave characterizations; forinstance p=3+6=9 for identifying P waves, QRS complexes, and T waves,and characterizing premature waves, paced waves, ventricular QRScomplexes, junctional QRS complexes, ectopic P waves and non-conducted Pwaves. According to said embodiment, annotations are expressed as listsof wave with their start and end points and characteristics such as forexample: (P, 1.2s, 1.3s, [non-conducted]), (QRS 1.4s 1.7s, [premature,ventricular]), (T, 1.7, 2.1), (P, 2.2, 2.3); in this example, the firstrow of y, corresponding to P waves, will be 1 for samples correspondingto times between 1.2 and 1.3s, and between 2.2 and 2.4s, and 0otherwise; row 2 will correspond to QRS complexes, row 3 to T waves, androw 4 corresponding to the premature characterization will be 1 duringthe premature QRS complex and 0 otherwise.

This invention also provides a method for manufacturing a neural networkfor the categorization of a cardiac signal, by training it.

In a multi-label classification, the manufacturing/training processincludes the following steps:

-   -   taking one cardiac signal from a dataset containing cardiac        signals and their known labels; the cardiac signal must be        expressed as a matrix of size m×n with m fixed and at a        predefined frequency;    -   expressing the labels as a vector of size q, with q the number        of labels to identify; this vector could be [0; 1; 0; 0; 1; 0;        0; 0] for q=8; a 1 is set in the vector at the index        corresponding to the labels which are present (i.e. having a        score above at least one predefined threshold such has for        instance 0.5): in the above example, the cardiac signal exhibits        two labels;    -   computing the output of the network for this cardiac signal;    -   modifying the parameters of the network so as to decrease a cost        function comparing the known label vector and the output of the        network; a cross-entropy error function is used so as to allow        for multi-labeling (allowing for multiple labels for a cardiac        signal): this minimization can be done though a gradient step;    -   repeating steps 1 to 4 at least once for each cardiac signal of        the dataset;    -   recovering the neural network.

This invention also provides a method for manufacturing a neural networkfor both the delineation and the categorization of a cardiac signal, bytraining it.

In the embodiment of the combination of delineation with multi-labelclassification, the manufacturing process includes the following steps:

-   -   taking one cardiac signal from a dataset containing cardiac        signals and their known labels: the cardiac signal must be        expressed as a matrix of size m×n with m fixed and at a        predefined frequency;    -   expressing the labels as a vector of size q, with q the number        of labels to identify; this vector could be [0; 1; 0; 0; 1; 0;        0; 0] for q=8; a 1 is set in the vector at the index        corresponding to the labels which are present (i.e. above a        predefined threshold): in the above example, the cardiac signal        exhibits two labels;    -   expressing the delineation of this cardiac signal under the form        of a matrix Y of size p×n where p is the number of waves to        identify; typically p=3, so as to identify P waves, QRS waves,        and T waves; annotations are usually expressed as lists wave        type with their start and end points such as for example: (P,        1.2s, 1.3s), (QRS 1.4s 1.7s), (T, 1.7, 2.1), (P, 2.2, 2.3); in        this example, the first row of Y, corresponding to P waves, will        be 1 for samples corresponding to times between 1.2 and 1.3s,        and between 2.2 and 2.4s, and 0 otherwise; row 2 will correspond        to QRS complexes and row 3 to T waves;    -   computing both outputs of the network for this cardiac signal;    -   modifying the parameters of the network so as to decrease the        sum of a cost function comparing the known label vector and one        of the output of the network, and a cost function comparing the        delineation and the other output; cross-entropy error functions        are used to allow for multi-labeling (allowing for multiple        labels for a cardiac signal as well as multiple waves at any        time point); this minimization can be done though a gradient        step;    -   repeating steps 1 to 4 at least once for each cardiac signal of        the dataset;    -   recovering the neural network.

According to one embodiment, the step of expressing the delineation ofthe cardiac signal under the form of a matrix Y of size p×n furthercomprises wave characterization. According to said embodiment, p is thenumber of annotated types of wave plus the number of wavecharacterizations; for instance p=3+6=9 for identifying P waves, QRScomplexes, and T waves, and characterizing premature waves, paced waves,ventricular QRS complexes, junctional QRS complexes, ectopic P waves andnon-conducted P waves. According to said embodiment, annotations areexpressed as lists of wave with their start and end points andcharacteristics such as for example: (P, 1.2s, 1.3s, [non-conducted]),(QRS 1.4s 1.7s, [premature, ventricular]), (T, 1.7, 2.1), (P, 2.2, 2.3);in this example, the first row of y, corresponding to P waves, will be 1for samples corresponding to times between 1.2 and 1.3s, and between 2.2and 2.4s, and 0 otherwise; row 2 will correspond to QRS complexes, row 3to T waves, and row 4 corresponding to the premature characterizationwill be 1 during the premature QRS complex and 0 otherwise.

This invention also pertains to a method and a device for delineation ofa cardiac signal, implementing a convolutional neural network,preferably a fully convolutional neural network, trained for delineationof a cardiac signal as described above.

As a basis, it shall be understood that the cardiac signal is expressedas a matrix X of size m×n at the frequency used for training thenetworks. The cardiac signal is used as input of the trained neuralnetwork.

The neural network then reads each time point of the cardiac signal,analyzes spatio-temporally each time point of the cardiac signal,assigns a temporal interval score to anyone of at least the following:P-wave, QRS complex, T-wave. It then recovers the output of the neuralnetwork, as a matrix Y of size p×n. An example is shown in FIG. 4: thefirst signal shows one of the leads of the ECG (to help thevisualization), the following 3 signals are the outputs of the network,providing scores for P waves, QRS waves and T waves. As it can be seen,these scores are synchronized with the appearance of the aforementionedwaves in the signal. According to one embodiment, when multiple leadscardiac signal is used, all the leads are processed simultaneously.

In a preferred embodiment, the neural network provides scores at eachtime point as a matrix Y, and a post-processing allows the allocation ofeach time point to none, single, or several waves, and provides theonset and offset of each of the identified waves as well as optionallyits characterization. For instance, a sample can be affected to thewaves for which the score on the corresponding row of Y is larger than0.5. Wave characterization such as conductivity, prematurity and originof the wave can be recovered from the activation of the correspondingrow between the onset and the offset of the wave. The premature labelcan for instance be applied to the wave if the average of the rowcorresponding to the premature characterization is above 0.5 during thewave. This provides a delineation sequence of type (P, 1.2s, 1.3s,[non-conducted]), (QRS 1.4s 1.7s, [premature, ventricular]), (T, 1.7s,2.1s), (P, 2.2s, 2.3s), as recorded in the annotations.

The invention also comprises a computer device implemented softwarecomprising a trained neural network for delineation of a cardiac signal.The invention also comprises a device, such as for example a cloudserver, a commercial ECG device, a mobile phone or a tablet, comprisinga software implementing the method for delineation as described above.

According to one embodiment, the device further comprises a displayconfigured for displaying the wave locations and optionallysimultaneously the cardiac signal.

According to one embodiment, global measurements derived from thedelineation sequence such as for instance the PR interval are displayed.According to one embodiment, global measurements derived from thedelineation sequence are highlighted for values which are not in anormal range. According to one embodiment, local measurements such asfor instance all RR intervals are displayed with the cardiac signal.According to one embodiment, the conduction pattern of the cardiacsignal is displayed in order to easily visualize characterization suchas for instance prematurity of the waves with the cardiac signal. In anembodiment, the waves are displayed according to time with the cardiacsignal.

This invention also pertains to a method and a device for multi-labelclassification of a cardiac signal, implementing Long-term RecurrentConvolutional Networks (LRCN. (Donahue et al., arXiv:1411.4389v3, 17Feb. 2015). These neural networks are trained for multi-labelclassification of a cardiac signal as described above.

As a basis, it shall be understood that the cardiac signal is expressedas a matrix of size m×n at the frequency used for training the networks.Then, the cardiac signal is used as input of the trained neural network.

The neural network then reads each time point of the cardiac signal,analyzes temporally each time point of the cardiac signal, computes ascore for each label, recovers the output of the neural network. In anembodiment, the labels are non-exclusive.

In an embodiment, some other information can be included as inputs ofthe network. Said information can be delineation-derived such as forinstance PR interval duration, heart rate, ST elevation or amplitudes ofthe QRS waves. It can also be patient-based such as their age or anyrelevant clinical information.

In an embodiment, the neural network NN2 reads and analyzes each timepoint of the cardiac signal and further the global measurements obtainedfrom NN1.

In a preferred embodiment, the neural network recovers the output as avector of size q. This vector contains scores for the presence of eachlabel. According to one embodiment, a label is considered as present ifits score is above a predefined threshold. This threshold is usually setto 0.5. It can however be modified to provide a differentsensitivity-specificity couple. Indeed, increasing the threshold leadsto lower specificity and higher specificity, and conversely whendecreasing it. This set of couples is called a receiver operatingcharacteristics curve and any point of this curve can be chosen througha modification of the threshold.

The invention also comprises a computer device implemented softwarecomprising a trained neural network for multi-label classification of acardiac signal. The invention also comprises a device, such as forexample a cloud server, a commercial ECG device, a mobile phone or atablet, comprising a software implementing the method of multi-labelclassification of a cardiac signal as described above.

According to one embodiment, the device further comprises a displayconfigured for displaying the scores of the labels which have beenallotted to a time window and optionally simultaneously the cardiacsignal.

According to an embodiment, the list of found labels for which the scorein the vector are higher than a predefined threshold, typically 0.5 isdisplayed. Labels can also be added depending on the delineation(delineation-based label), such as for instance the label correspondingto first degree atrioventricular block which is equivalent to a PRinterval longer than 200 ms, said PR interval being a global measurementbased on the delineation. The list of labels can finally be filtered toremove redundant labels based on a known hierarchy of labels (forinstance only the most detailed labels are retained), or aggregatedthrough time on long cardiac signal so as to recover the start and endtimes of each abnormality.

This invention also pertains to a method and a device for delineationand multi-label classification of a cardiac signal, implementing aneural network trained for delineation and multi-label classification ofa cardiac signal as described above.

As a basis, it shall be understood that the cardiac signal is expressedas a matrix of size m×n at the frequency used for training the networks.Then, the cardiac signal is used as input of the trained neural network.

The neural network then reads each time point of the cardiac signal,analyzes temporally each time point, assigns a temporal score to all ofthe following at least: P-wave, QRS complex, T-wave. It then computes ascore for each labels, recovers both the outputs of the neural network:the first as a matrix y of size p×n, providing scores for at least Pwaves, QRS waves and T waves; and the second as a vector of size q, saidvector containing scores for the presence of each label.

In a preferred embodiment, a post-processing of the delineation outputallows to affect each time point to none, single, or several waves, andprovides the onset and offset of each of the identified waves. Forinstance, a sample can be affected to the waves for which the score onthe corresponding row of Y is larger than 0.5. This provides adelineation sequence of type (P, 1.2s, 1.3s), (QRS 1.4s 1.7s), (T, 1.7s,2.1s), (P, 2.2s, 2.3s), as recorded in the annotations.

According to an embodiment, the list of found labels for which the scorein the vector are higher than a predefined threshold, typically 0.5, aredisplayed; as well as the delineation, optionally with the cardiacsignal.

According to an embodiment of the invention, a step to prepare thesignal and create input variables for classification is further carriedout (“pre-treatment”). The purpose of this pre-treatment is to removethe disturbing elements of the signal such as for example noise andbaseline, low frequency signal due to respiration and patient motion, inorder to facilitate classification. For noise filtering, a multivariateapproach functional analysis proposed by (Pigoli and Sangalli,Computational Statistics and Data Analysis, vol. 56, 2012, pp 1482-1498)can be used. The low frequencies of the signal corresponding to thepatient's movements may be removed using median filtering as proposed by(Kaur et al., Proceedings published by International Journal of ComputerApplications, 2011, pp 30-36).

According to an embodiment of the invention, a post-treatment step isadded, so as to produce the onset and offset of each wave in the cardiacsignal.

The invention also comprises a computer device implemented softwarecomprising a trained neural network for delineation and multi-labelclassification of a cardiac signal. The invention also comprises adevice, such as for example a cloud server, a commercial ECG device, amobile phone or a tablet, comprising a software implementing the methodof delineation and multi-label classification of a cardiac signal asdescribed above.

According to one embodiment, the device further comprises a displayconfigured for displaying the wave locations, the scores of the labelswhich have been allotted to a time window and optionally simultaneouslythe cardiac signal.

In an embodiment, global and local measurements derived from thedelineation sequence such as for instance the PR interval are displayed.In an embodiment, the global and local measurements derived from thedelineation sequence are highlighted for values which are not in anormal range. In an embodiment, the conduction pattern of the cardiacsignal is displayed in order to easily visualize characterization suchas for instance prematurity of the waves; and the waves may be displayedaccording to time.

The present invention further relates to a system comprising anelectrocardiograph for recording cardiac signal and for implementing themethods according to the present invention. Thus, the electrocardiographprovides labels, delineation, measurements and conduction pattern of thecardiac signal right after the recording.

This invention brings to the art a number of advantages, some of thembeing described below:

-   -   The input of the networks are one or multi-lead cardiac signals        with variable length, possibly preprocessed so as to remove        noise and baseline wandering due to patients movements, and        express the signal at a chosen frequency.    -   Using the presented lead invariant structure, a same network can        handle cardiac signals with different number of leads.    -   The output of a classification network is a vector of scores for        labels. These are not classification scores since one cardiac        signal can present several labels. For example, the output of        such network could be a vector [0.98; 0.89; 0.00; . . . ] with        the corresponding labels for each element of the vector (Right        Bundle Branch Bloc; Atrial Fibrillation; Normal ECG; . . . ).        Scores are given between a scale of [0, 1] and the above example        output vectors therefore indicates a right bundle branch block        and atrial fibrillations. A recurrent neural network        architecture can be added on the top of the convolutional        network (Donahue et al., arXiv:1411.4389v3, 17 Feb. 2015 and        Mnih et al., arXiv:1406.6247v1, 24 Jun. 2014). In this way, the        convolution network acts as a pattern detector whose output will        be accumulated in time by the recurrent network.    -   The output of the delineation network is a set of signals        spanning the length of the input cardiac signal, providing the        score for being in waves such as for instance P waves, QRS        complexes, a T waves and potentially other types of waves or        segments such as for example flutter waves, U waves or noisy        segments. An example of output signals is provided in FIG. 5.    -   The delineation network can also characterize the waves such as        for instance their prematurity, conductivity and ectopy. This        ability allows to unify two steps of the delineation which are        separate in current methods, making the proposed method more        reliable and able to leverage context information for the waves        characterization.    -   The delineation network is not limited to recovering at most one        wave at each time point and therefore can identify several waves        at any time point, such as for instance P waves hidden in a T        wave.    -   The delineation network allows both the recovery of the start        and end of each wave and there characterization in a single        step, which is more reliable than all previous methods. In        particular in is more reliable to recover P waves and        characterize them, allowing to provide the conductivity pattern        of the cardiac signal.    -   No works applying convolutional networks to the delineation have        been made so far.

The underlying structure of the networks is not fundamental as long asthey are convolutional neural networks. One can use a structure such asRLCN (Donahue et al., arXiv:1411.4389v3, 17 Feb. 2015 and Mnih et al.,arXiv:1406.6247v1, 24 Jun. 2014) for classification and a networksimilar as the one in (Long et al., Proceedings of Computer Vision andPattern Recognition, 2015, pp 3431-3440) for delineation. In bothembodiments, convolutional layers must be modified as 1D convolutionsinstead of 2D convolutions. On top of these architectures, bothembodiments can use a lead invariant structure such as but not limitedto the one presented in FIG. 8.

A hybrid network, sharing the first layers and diverging so as toprovide both the delineation as one output, and the multi-labelclassification as another output is also used. This combination has theadvantage of being able to produce a multi-label classification helpedby the identification of the cardiac signal waves.

EXAMPLES

The neural networks used within the present invention, were filed atLOGITAS under number D16201.

The present invention is further illustrated by the following examples.

Example 1: Training for Delineation

This training was performed on 2204 ECGs and the network evaluated onabout 900 beats from 77 different patients which were not used for thetraining phase. The following table provides the precision of the waveonsets (beginnings) and offsets (ends) in term of bias and standarddeviation (std) as well as the false positive (FP) and false negative(FN) rates of the waves detection and of their characterizations:

EP (%) EN (%) Bias (ms) Std (ms) Count P 7.9 5.6 0.9 10.6 730 PQ 0 0.30.3 7.2 616 QRS 0 0 1.8 5.2 887 QT 0 0.1 0.7 13.3 873 P onset N/A N/A−2.9 6.4 689 P offset N/A N/A −2 8.4 689 QRS onset N/A N/A −3.1 4 887QRS offset N/A N/A −1.3 3.6 887 QT offset N/A 7 −2.3 12.3 872 ectopic P6 16 N/A N/A 75 premature P 0 20 N/A N/A 10 paced P 0 24.3 N/A N/A 37non-conducted P 0 8.3 N/A N/A 36 ventricular QRS 7.7 5.3 N/A N/A 38premature QRS 8.3 15.4 NIA N/A 26 paced QRS 0 0 N/A N/A 22 junctionalQRS 0 14.6 N/A N/A 48

Concerning hidden P waves, the proposed algorithm was able to recover 75out of 87 hidden P waves present in this evaluation dataset, while otheralgorithms would not be able to find any of them.

From the onsets and offsets of each wave are derived standard globalmeasurements such as the P duration, PR interval, QRS duration and QTinterval. An evaluation was performed on the standard CSE dataset whichprovides acceptance limits for delineation algorithms (Christov et al.BioMedical Engineering OnLine, 2006, vol. 5, pp. 31-38), yielding thefollowing results which are well within the acceptance range:

Standard Deviation (ms) Bias (ms) Measurement Result Limit Result LimitP 3.8 15 −2.4 10 PQ 6.1 10 0.1 10 QRS 4.2 10 2.0 10 QT 7.2 30 −14.2 25

The following table sums up the results on the MIT-BIH ArrhythmiaDatabase (Moody et al, Computers in Cardiology, 1990, vol. 17, pp.185-188) of a delineation network with a lead-invariant structure, whichwas not used for the training, in terms of QRS and premature ventricularcomplexes (PVC) detections:

FP (%) FN (%) Count QRS 0.32 0.17 107341 PVC 7.68 15.10 7071

Compared with state-of-the-art algorithms, the precision was improvedand the ability of the algorithm, which can find the waves andcharacterize them at the same time, is much more efficient. In FIG. 5for instance, the ECG exhibits an atrioventricular block which meansthat the P waves and the QRS complexes are completely decoupled. In thisexample, the P waves are regular and QRS complexes happen at randomtimes. One can observe in this example that the algorithm correctlyfound two P waves between the first QRS complex and the second QRScomplex, while most algorithms would not be able to find them since theylook for only one P wave before each complex. The last P wave alsostarts before the end of the last T wave, adding complexity. Finally,the algorithm is able to characterize theses waves as non-conducted.Other algorithms would not have been able to find the hidden waves andwould not have been able to characterize any wave as non-conducted.

Example 2: Training for Multi-Label Classification

A network has been trained using about 85,000 ECGs and has beenevaluated on a dataset representative of a hospital emergency unitincluding 1,000 patients which were not used in the training phase. Theresults in terms of accuracy, specificity, sensitivity, and positivepredict values were the following for some of the searched labels:

Population Accuracy Sensitivity Specificity PPV Normal ECG 421  77.39% 66.75%  89.06%  87.00% Atrial fibrillation 22  99.75%  90.91% 100.00%100.00% Atrial flutter 3  99.88%  66.67% 100.00% 100.00% Junctionalrhythm 5  99.75%  80.00%  99.88%  80.00% Pacemaker 5 100.00% 100.00%100.00% 100.00% Premature 17  99.63%  88.24%  99.87%  93.75% ventricularcomplex(es)         Complete 21  99.50%  90.48%  99.74%  90.48% rightbundle branch block         Complete 4  99.75%  75.00%  99.88%  75.00%left bundle branch block         Left ventricular  99.38%  57.14% 99.75%  66.67% hypertrophy Acute STEW 5 100.00% 100.00% 100.00% 100.00%Old MI 27  93.79%  70.37%  94.60%  31.15%

A neural network with a lead-invariant structure aimed at classifyingrhythm abnormalities was also trained. Its performance on Holter ECGs interm of atrial fibrillation was analyzed on the MIT-BIH ArrhythmiaDatabase (Moody et al, Computers in Cardiology, 1990, vol. 17, pp.185-188) comprising 30 minutes 2-lead ECGs of 48 different patients. Tothis end, the neural networks analyzed all 20 second segments of theECG, which providing a rhythm label each 20 second, which wereaggregated to provide the beginning and end of each rhythm abnormalityor descriptor. The recovered labels were compared to the referenceannotations, yielding a, accuracy, sensitivity, specificity and positivepredictive value (PPV) for the atrial fibrillation label and the lessspecific atrial fibrillation or flutter label:

Accuracy Sensitivity Specificity PPV Atrial fibrillation 98.3% 96.9%98.5% 89.6% Atrial fibrillation or flutter 99.0% 96.8% 99.2% 92.3%

These results are similar to the state-of-the-art in term ofsensitivity, but significantly better than state-of-the-art methods interm of specificity and therefore also in accuracy and PPV.

A graphical representation of how a standard multi-label is used on ECGsis displayed in FIG. 6. The EGG is given as input to the network, whichaggregates the information locally and then combines it layer by layerto produce a high-level multi-label classification of the ECG, in thisexample correctly recognizing atrial fibrillations and a right bundlebranch block. Such networks however take a fixed size as input and theprocess must be reproduced at different locations so as to analyze thewhole signal. FIG. 7 is an example of a graphic representation of arecurrent neural network which overcomes this issue. This type ofnetwork is made from a standard convolutional network computed at allpossible locations of the signal, and on top of which comes anothernetwork layer which accumulates the information. In this example, thenetwork correctly recognizes a premature ventricular complex (PVC, thefifth and largest beat) in the first part of the signal while the secondpart of the signal is considered as normal. The accumulated output istherefore PVC since this ECG has an abnormality and cannot therefore beconsidered as normal.

Example 3: Delineation and Multi-Label Classification

In another embodiment, the applicant combines features described abovein examples 1 and 2. Such combination enables to combine the advantagesof both networks in a unique network, providing similar results for boththe delineations and the multi-label classifications.

Example 4: Platform Use Case

According to one embodiment, a user can log into a web platform. Anupload button is available for the user to upload one of their ECGs in asupported formal so as to process it. The user is then redirected to apage displaying the ECG as shown in FIG. 9. The user can than select toshow the detected abnormalities as shown in FIG. 10. In this case, theneural networks correctly detected a second degree atrioventricularblock Mobitz I (a lengthening of the PR interval leading to anon-conducted P wave) and an intraventricular block (causing alengthened QRS duration). The user can also choose to display thedelineation information as shown in FIG. 11. This information includeshighlighting of the identified waves on the ECG drawing, printing globalmeasurements derived from the delineation above the ECG drawing such asfor example heart rate (HR) and QRS duration (QRS), with highlighting ofthe values which are not in a normal range such as for instance a QRSduration larger than 110 ms. Local measurements such as all RR intervalsare also shown as figures under the ECG. It also includes aladdergram-like feature showing the conduction pattern of the waves,their prematurity and their origin. This feature is displayed under theECG, with each dot on the first line being a P wave, each dot of thesecond wave being a QRS wave and lines between them implying conduction.One can observe in this case that some P waves are not conducted. Thesesame P waves are hidden P waves since they occur during T waves. InFIGS. 12 and 13, one can see different examples of the conductionpattern display where the prematurity of the P and QRS waves are shown(with surrounding squares), and the origin of the waves are also shown(lightning bolt for a paced wave, square for ectopic P wave orventricular QRS complex, triangle for junctional QRS complex etc.).

Example 5: Application Programming Interface (API) Use Case

According to an embodiment, a user can also send an ECG through an API.The ECG is received on the platform and analyzed. The user can thenrecover information such as the delineation and the multi-labelclassification through another API.

Example 6: Resting ECG Interpretation

A patient arrives at the emergency unit of a hospital and an ECG isperformed. The ECG shows wide complex tachycardia. Such a pattern canoccur in very different situations, such as in the case of ventriculartachycardia, or with both atrial fibrillation and Wolff-Parkinson-Whitesyndrome, or with both a bundle branch block and sinus tachycardia. Suchconditions must be treated differently, the two former beinglife-threatening. Standard algorithms of the prior art can only detectone abnormality at a time and not a combination of labels. In this case,it is however crucial to be able to perform multi-label classificationsince interpretations may imply a combinations of labels. Being able todo so help properly identifying an actual ventricular tachycardia thatother algorithms have difficulty to identify such as the one in FIG. 14Indeed, method according to the present invention is able to test allother possible combinations of labels and rule them out.

Also, during an examination a general practitioner performs an ECG on apatient. The delineation is then helpful in order to highlight hidden Pwaves which may completely change the diagnostic between normal sinusrhythm and a 2^(nd) degree atrioventricular block which may require theuse of a pacemaker.

Example 7: Holter Interpretation

A patient is prescribed a 7 day Holter. The 7 days must afterwards beinterpreted by a specialist. The proposed algorithm is able to identifynoisy segments of the signal which are common in Holters since thepatient is allowed to move. It can also find atrial fibrillation oratrial flutter which is often looked at in Holters. Thanks to itsmulti-label ability, the proposed algorithm can also find atrialfibrillation during noise segments. In other situations, the patientcould be monitored at a hospital in order to assess the possibility ofan acute myocardial infarction. The proposed method can then provide STelevations through time thanks to the delineation (amplitude at the QRSoffset minus amplitude at the QRS onset) which changes are a veryimportant indicator of STEMI (ST elevation myocardial infarction).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a photo of an ECG.

FIG. 2 is a schematic representation of a normal ECG, with the P wave,the QRS complex/wave comprising the Q, R, S and J points, and the Twave.

FIG. 3 is an example of structure for a basic neural network with noconvolutional layer.

FIG. 4 is an example of the output of the delineation network on anormal ECG.

FIG. 5 is an example of the output of the delineation network on an ECGwith hidden P waves (high degree atrioventricular block).

FIG. 6 models the way a standard multi-label convolutional networkworks.

FIG. 7 models the way a multi-label recurrent convolutional networkworks.

FIG. 8 provides an example of structure to use as first layers of aneural network to make it able to process any number of leads.

FIG. 9 shows the interface after loading an ECG.

FIG. 10 shows the interface with the ECG and the labels provided by themulti-label classification algorithm.

FIG. 11 shows the interface with the ECG, the labels provided by themulti-label classification algorithm, the delineation of the waves onthe ECG, measurements derived from the delineation above the ECG withhighlighted abnormal values, and conduction pattern under the ECGderived from the delineation. Some waves, including one hidden P waveare more specifically shown.

FIG. 12 shows an example ECG with its conduction pattern comprisingpremature atrial complexes and one paced P waves. This conductionpattern is composed of a first line with the P waves as dots, and asecond line with the QRS as dots. It can be synchronized or not with thesignal depending on its use (not synchronized on this example).

FIG. 13 shows an example ECG with its conduction pattern comprisingnon-conducted P waves, paced QRS complexes and premature ventricularcomplexes. These elements are pointed at on the figure.

FIG. 14 shows an ECG with a ventricular tachycardia which was notidentified by a prior art algorithm but is correctly identified by theproposed algorithm.

What is claimed is:
 1. A system for detecting anomalies in anelectrocardiogram (ECG) signal corresponding to a patient, the systemcomprising at least one server and at least one processor configured toexecute instructions to: receive the ECG signal sampled from the patientover a plurality of time points; determine delineation scores at eachtime point of the plurality of time points of at least a portion of theECG signal, the delineation scores corresponding to multiple wave typesat each time point; determine anomaly scores based on the at least aportion of an ECG signal and the delineation scores, the anomaly scorescorresponding to multiple anomaly types; determine that an anomalycorresponding to one of the multiple anomaly types is present in the ECGsignal based on at least one of the anomaly scores corresponding to afirst time point of the plurality of time points; and generateinformation for display comprising a graphical representation of the atleast a portion of the ECG signal and a visual indication of the anomalycorresponding to the graphical representation of the at least a portionof the ECG signal at the first time point of the plurality of timepoints.
 2. The system of claim 1, wherein determining that the anomalyis present in the ECG signal comprises determining that atrialfibrillation is present in the ECG signal.
 3. The system of claim 1,wherein the visual indication of the anomaly is aligned with a locationof the anomaly on the graphical representation of the at least a portionof the ECG signal.
 4. The system of claim 1, wherein the at least oneprocessor is further configured to execute instructions to: determinethat a second anomaly corresponding to a second one of the multipleanomaly types and different from the first anomaly is present in the ECGsignal based on at least a second one of the anomaly scorescorresponding to a second time point of the plurality of time points;and generate information for display comprising a second visualindication of the second anomaly corresponding to the graphicalrepresentation of the at least a portion of the ECG signal at the secondtime point of the plurality of time points.
 5. The system of claim 1,wherein generating information for display comprises generatingdelineation information highlighting on the graphical representation ofthe at least a portion of the ECG signal at least one wave type of themultiple wave types.
 6. The system of claim 1, wherein the at least oneprocessor is further configured to execute instructions to cause displayof the graphical representation of the at least a portion of the ECGsignal and the visual indication of the anomaly.
 7. The system of claim1, wherein the at least one processor is further configured to executeinstructions to determine a threshold value indicative of the presenceof the anomaly.
 8. The system of claim 7, wherein determining that theanomaly is present in the ECG signal comprises comparing the anomalyscores to the threshold value and determining that the at least one ofthe anomaly scores satisfies the threshold value.
 9. The system of claim1, wherein determining delineation scores comprises applying the atleast a portion of the ECG signal to a first neural network.
 10. Thesystem of claim 9, wherein determining anomaly scores comprises applyingthe at least a portion of the ECG signal and the delineation scores to asecond neural network.
 11. A computerized method for detecting anomaliesin an electrocardiogram (ECG) signal corresponding to a patient, thecomputerized method comprising: receiving an ECG signal sampled from apatient over a plurality of time points; determining delineation scoresat each time point of the plurality of time points of at least a portionof the ECG signal, the delineation scores corresponding to multiple wavetypes at each time point; determining anomaly scores based on the atleast a portion of an ECG signal and the delineation scores, the anomalyscores corresponding to multiple anomaly types; determining that ananomaly corresponding to one of the multiple anomaly types is present inthe ECG signal based on at least one of the anomaly scores correspondingto a first time point of the plurality of time points; and generatinginformation for display comprising a graphical representation of the atleast a portion of the ECG signal and a visual indication of the anomalycorresponding to the graphical representation of the at least a portionof the ECG signal at the first time point of the plurality of timepoints.
 12. The computerized method of claim 11, wherein determiningthat the anomaly is present in the ECG signal comprises determining thatatrial fibrillation is present in the ECG signal.
 13. The computerizedmethod of claim 11, wherein the visual indication of the anomaly isaligned with a location of the anomaly on the graphical representationof the at least a portion of the ECG signal.
 14. The computerized methodof claim 11, further comprising: determining that a second anomalycorresponding to a second one of the multiple anomaly types anddifferent from the first anomaly is present in the ECG signal based onat least a second one of the anomaly scores corresponding to a secondtime point of the plurality of time points; and generating informationfor display comprising a second visual indication of the second anomalycorresponding to the graphical representation of the at least a portionof the ECG signal at the second time point of the plurality of timepoints.
 15. The computerized method of claim 11, wherein generatinginformation for display comprises generating delineation informationhighlighting on the graphical representation of the at least a portionof the ECG signal at least one wave type of the multiple wave types. 16.The computerized method of claim 11, further comprising causing displayof the graphical representation of the at least a portion of the ECGsignal and the visual indication of the anomaly.
 17. The computerizedmethod of claim 11, further comprising determining a threshold valueindicative of the presence of the anomaly.
 18. The computerized methodof claim 17, wherein determining that the anomaly is present in the ECGsignal comprises comparing the anomaly scores to the threshold value anddetermining that the at least one of the anomaly scores satisfies thethreshold value.
 19. The computerized method of claim 11, whereindetermining delineation scores comprises applying the at least a portionof the ECG signal to a first neural network.
 20. The computerized methodof claim 19, wherein determining anomaly scores comprises applying theat least a portion of the ECG signal and the delineation scores to asecond neural network.
 21. A programmed routine for use with acomputerized system for detecting abnormalities in an electrocardiogram(ECG) signal obtained from a patient, the programmed routine comprisinginstructions that when executed: receive an ECG signal sampled from apatient over a plurality of time points; determine delineation scores ateach time point of the plurality of time points of at least a portion ofthe ECG signal, the delineation scores corresponding to multiple wavetypes at each time point; determine anomaly scores based on the at leasta portion of an ECG signal and the delineation scores, the anomalyscores corresponding to multiple anomaly types; determine that ananomaly corresponding to one of the multiple anomaly types is present inthe ECG signal based on at least one of the anomaly scores correspondingto a first time point of the plurality of time points; and generateinformation for display comprising a graphical representation of the atleast a portion of the ECG signal and a visual indication of the anomalycorresponding to the graphical representation of the at least a portionof the ECG signal at the first time point of the plurality of timepoints.
 22. The programmed routine of claim 21, wherein determining thatthe anomaly is present in the ECG signal comprises determining thatatrial fibrillation is present in the ECG signal.
 23. The programmedroutine of claim 21, wherein the visual indication of the anomaly isaligned with a location of the anomaly on the graphical representationof the at least a portion of the ECG signal.
 24. The programmed routineof claim 21, wherein the programmed routine further comprisesinstructions that when executed: determine that a second anomalycorresponding to a second one of the multiple anomaly types anddifferent from the first anomaly is present in the ECG signal based onat least a second one of the anomaly scores corresponding to a secondtime point of the plurality of time points; and generate information fordisplay comprising a second visual indication of the second anomalycorresponding to the graphical representation of the at least a portionof the ECG signal at the second time point of the plurality of timepoints.
 25. The programmed routine of claim 21, wherein generatinginformation for display comprises generating delineation informationhighlighting on the graphical representation of the at least a portionof the ECG signal at least one wave type of the multiple wave types. 26.The programmed routine of claim 21, wherein the programmed routinefurther comprises instructions that when executed cause display of thegraphical representation of the at least a portion of the ECG signal andthe visual indication of the anomaly.
 27. The programmed routine ofclaim 21, wherein the programmed routine further comprises instructionsthat when executed determine a threshold value indicative of thepresence of the anomaly.
 28. The programmed routine of claim 27, whereindetermining that the anomaly is present in the ECG signal comprisescomparing the anomaly scores to the threshold value and determining thatthe at least one of the anomaly scores satisfies the threshold value.29. The programmed routine of claim 21, wherein determining delineationscores comprises applying the at least a portion of the ECG signal to afirst neural network.
 30. The programmed routine of claim 29, whereindetermining anomaly scores comprises applying the at least a portion ofthe ECG signal and the delineation scores to a second neural network.